chore: add APM integration to CI by mdelapenya · Pull Request #1148 · elastic/e2e-testing

mdelapenya · 2021-05-06T09:48:01Z

What does this PR do?

It adds a parallel execution running the tests for the APM Integration. Besides that, because this PR was outdated with the codebase, I'm doing the following tasks:

update the test scenarios with latest versions of the steps
remove fleet server bootstrapping from the stand-alone scenarios, as it's already provided by the runtime dependencies (starting the Fleet test suite implies starting the stack + fleet-server).
decouple operations for Fleet integrations into a separate feature file. This way we will be able to differentiate potential errors related to CRUD operations in the integrations from errors. For that, we are moving the operations to the new feature file, removing them from the original scenarios. Therefore, these scenarios will be simpler, testing one specific behaviour.
minor refactors related to removing leftovers or renaming scenarios for consistency
properly enroll standalone agent into Fleet (cc/ @EricDavisX @adam-stokes @michalpristas): we discovered that the standalone agent never enrolled in Fleet, and the the stand-alone agent is listed in Fleet as "online" step was getting the online status for the wrong agent (the first agent in the list, which was the already bootstrapped fleet-server agent). For that reason we are getting the hostname of current agent and checking that it appears in the response of the agents list. If so, we verify that it's in the online status. The auto-enrollment has been implemented with ce45452, where the proper env vars have been declared and populated so the agent enrolls in Fleet using the already bootstrapped fleet-server.

Why is it important?

We were not running those tests on CI

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests for the CLI, and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

@jalvz, we are running the tests in parallel on CI, can you confirm that this is what we want for those tests?
@elastic/apm-server as discussed offline, we'd need your help in fixing these tests.

How to test this PR locally

TAGS="apm_server && install" TIMEOUT_FACTOR=5 LOG_LEVEL=TRACE DEVELOPER_MODE=true ELASTIC_APM_ACTIVE=false make -C e2e/_suites/fleet functional-test

Related issues

Completes apm-server tests #1083

Backports

Backport to 7.x, and check if 7.14.x and 7.13.x are required

elasticmachine · 2021-05-06T10:17:09Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-07-28T14:53:56.404+0000
Duration: 37 min 43 sec
Commit: 82393c3

Test stats 🧪

Test	Results
Failed	0
Passed	240
Skipped	0
Total	240

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	240
Skipped	0
Total	240

mdelapenya · 2021-05-06T10:20:55Z

Ok the APM tests are failing on CI. @jalvz could you take a look before merging this PR?

jalvz

oh i didn't know new tests did not run on CI... do we need also the cloud tag?

mdelapenya · 2021-05-06T13:13:00Z

oh i didn't know new tests did not run on CI... do we need also the cloud tag?

No, because the @apm_server tag defines the scope for the entire feature file. We usually use that feature tag for splitting scenarios on CI, using the descriptor modified in this PR

adam-stokes · 2021-07-12T16:44:54Z

/test

mdelapenya · 2021-07-19T07:07:43Z

@adam-stokes I'm currently working in this branch, as there are some outdated scenarios that need rework

mdelapenya · 2021-07-19T08:49:00Z

@jalvz sorry for taking that much in pushing this forward. I've just returned from vacations and would like to merge this ASAP.

I've found one issue with the current implementation of the aStandaloneAgentIsDeployed test method: it never achieves the apm-server process to be spun up in the agent. On the contrary, mounting the config files for "apm-legacy" as a volume (exactly the same as in the cloud scenario), then the process appears in the agent container. And that effect would lead to having exactly two equal scenarios.

Do you know why?

mdelapenya · 2021-07-20T10:07:47Z

Updated the description and pinged @elastic/apm-server team so that they are aware of how to reproduce this failure for the APM integration. As mentioned on Slack, they are testing the Fleet integration on every APM Server's PR, so they pointed out to a possible problem in tests' setup.

The installation of the integration to the policy is already tested in another scenario

We were getting the 1st agent, and because we have an agent bootstrapped as fleet-server, it was retrieved as the first one, causing that the "is agent 'online'" step always returned true, instead of returning the status of the newly deployed agent.

mdelapenya · 2021-07-27T15:56:16Z

Thousand thank yous to @michalpristas and @axw for the quick chats we had to unblock this.

* master: fix(ci): wait for downstream jobs on scheduled jobs (elastic#1393) Fix for the OS field and scenarios (elastic#1349) Add test case for add_kubernetes_metadata with autodiscover (elastic#1385) chore(ci): propagate downstream build result to upstream (elastic#1377)

mdelapenya · 2021-07-28T15:35:57Z

 Scenario Outline: Starting the <image> agent starts backend processes
  When a "<image>" stand-alone agent is deployed
-  Then there are "1" instances of the "filebeat" process in the "started" state
+  Then there are "2" instances of the "filebeat" process in the "started" state


This was an error hidden by the non-autoenrolling stand-alone agent.

* chore: add APM integration to CI * fix: update scenario step to latest version * chore: remove blank lines * fix: remove fleet server from the stand-alone agent * feat: add an scenario for adding integrations * chore: simplify scenarios avoiding testing twice The installation of the integration to the policy is already tested in another scenario * chore: rename scenario * chore: add integrations feature file to the CI * chore: remove references to FleetServerPolicy, as it's not used anymore * fix: bring fleet-server boostrap test back * fix: expose cloud agent in a not used port * chore: extract a method to get Fleet Server URL * fix: get stand-alone agent by hostname from agents list We were getting the 1st agent, and because we have an agent bootstrapped as fleet-server, it was retrieved as the first one, causing that the "is agent 'online'" step always returned true, instead of returning the status of the newly deployed agent. * fix: automatically enroll the stand-alone agent in Fleet * fix: reduce the number of occurrences * fix: there are 2 filebeat instances * chore: move cloud configs to a better place * chore: run APM tests with ubi8 base image * fix: keep original structure * fix: right volume path * chore: remove cloud scenario for APM integration (cherry picked from commit bc65335)

* chore: add APM integration to CI * fix: update scenario step to latest version * chore: remove blank lines * fix: remove fleet server from the stand-alone agent * feat: add an scenario for adding integrations * chore: simplify scenarios avoiding testing twice The installation of the integration to the policy is already tested in another scenario * chore: rename scenario * chore: add integrations feature file to the CI * chore: remove references to FleetServerPolicy, as it's not used anymore * fix: bring fleet-server boostrap test back * fix: expose cloud agent in a not used port * chore: extract a method to get Fleet Server URL * fix: get stand-alone agent by hostname from agents list We were getting the 1st agent, and because we have an agent bootstrapped as fleet-server, it was retrieved as the first one, causing that the "is agent 'online'" step always returned true, instead of returning the status of the newly deployed agent. * fix: automatically enroll the stand-alone agent in Fleet * fix: reduce the number of occurrences * fix: there are 2 filebeat instances * chore: move cloud configs to a better place * chore: run APM tests with ubi8 base image * fix: keep original structure * fix: right volume path * chore: remove cloud scenario for APM integration (cherry picked from commit bc65335) Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

mdelapenya self-assigned this May 6, 2021

mdelapenya requested review from a team and jalvz May 6, 2021 09:48

mdelapenya marked this pull request as ready for review May 6, 2021 09:48

jalvz reviewed May 6, 2021

View reviewed changes

v1v approved these changes May 10, 2021

View reviewed changes

cachedout approved these changes May 10, 2021

View reviewed changes

adam-stokes added v7.13.0 v7.14.0 labels Jul 12, 2021

mdelapenya added the v7.15.0 label Jul 19, 2021

mdelapenya added 7 commits July 27, 2021 17:45

chore: add APM integration to CI

6814e34

fix: update scenario step to latest version

194accc

chore: remove blank lines

ea12af8

fix: remove fleet server from the stand-alone agent

f050d37

feat: add an scenario for adding integrations

5d3c543

chore: simplify scenarios avoiding testing twice

5039a13

The installation of the integration to the policy is already tested in another scenario

chore: rename scenario

ec96be9

mdelapenya added 10 commits July 27, 2021 17:45

chore: add integrations feature file to the CI

7f57a22

chore: remove references to FleetServerPolicy, as it's not used anymore

f7bcf04

fix: bring fleet-server boostrap test back

bb518d7

fix: expose cloud agent in a not used port

60aebe0

chore: extract a method to get Fleet Server URL

0db8620

fix: automatically enroll the stand-alone agent in Fleet

ce45452

fix: reduce the number of occurrences

7fa5f42

fix: there are 2 filebeat instances

ff86a5e

chore: move cloud configs to a better place

c48752c

mdelapenya force-pushed the apm-server-tests branch from 161ff54 to c48752c Compare July 27, 2021 15:49

mdelapenya added 5 commits July 28, 2021 08:37

chore: run APM tests with ubi8 base image

69d866b

fix: keep original structure

679769c

fix: right volume path

d42e255

chore: remove cloud scenario for APM integration

82393c3

mdelapenya commented Jul 28, 2021

View reviewed changes

mdelapenya merged commit bc65335 into elastic:master Jul 28, 2021

mergify Bot mentioned this pull request Jul 28, 2021

[7.x](backport #1148) chore: add APM integration to CI #1397

Merged

mergify Bot mentioned this pull request Jul 28, 2021

[7.14.x](backport #1148) chore: add APM integration to CI #1398

Merged

mergify Bot mentioned this pull request Jul 28, 2021

[7.13.x](backport #1148) chore: add APM integration to CI #1399

Merged

mdelapenya deleted the apm-server-tests branch July 30, 2021 12:07

mdelapenya mentioned this pull request Oct 26, 2021

chore: bump elastic-package to v0.25.0 #1721

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add APM integration to CI#1148

chore: add APM integration to CI#1148
mdelapenya merged 22 commits intoelastic:masterfrom
mdelapenya:apm-server-tests

mdelapenya commented May 6, 2021 •

edited

Loading

Uh oh!

elasticmachine commented May 6, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

Uh oh!

mdelapenya commented May 6, 2021

Uh oh!

jalvz left a comment

Uh oh!

mdelapenya commented May 6, 2021

Uh oh!

adam-stokes commented Jul 12, 2021

Uh oh!

mdelapenya commented Jul 19, 2021

Uh oh!

mdelapenya commented Jul 19, 2021

Uh oh!

mdelapenya commented Jul 20, 2021

Uh oh!

mdelapenya commented Jul 27, 2021

Uh oh!

mdelapenya Jul 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

mdelapenya commented May 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Backports

Uh oh!

elasticmachine commented May 6, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

Uh oh!

mdelapenya commented May 6, 2021

Uh oh!

jalvz left a comment

Choose a reason for hiding this comment

Uh oh!

mdelapenya commented May 6, 2021

Uh oh!

adam-stokes commented Jul 12, 2021

Uh oh!

mdelapenya commented Jul 19, 2021

Uh oh!

mdelapenya commented Jul 19, 2021

Uh oh!

mdelapenya commented Jul 20, 2021

Uh oh!

mdelapenya commented Jul 27, 2021

Uh oh!

mdelapenya Jul 28, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mdelapenya commented May 6, 2021 •

edited

Loading

elasticmachine commented May 6, 2021 •

edited

Loading